{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-colab"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/pathwaycom/pathway/blob/main/examples/notebooks/tutorials/rag-evaluations.ipynb\" target=\"_parent\"><img src=\"https://pathway.com/assets/colab-badge.svg\" alt=\"Run In Colab\" class=\"inline\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "notebook-instructions",
      "source": [
        "# Installing Pathway with Python 3.10+\n",
        "\n",
        "In the cell below, we install Pathway into a Python 3.10+ Linux runtime.\n",
        "\n",
        "> **If you are running in Google Colab, please run the colab notebook (Ctrl+F9)**, disregarding the 'not authored by Google' warning.\n",
        "> \n",
        "> **The installation and loading time is less than 1 minute**.\n"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "id": "pip-installation-pathway",
      "source": [
        "%%capture --no-display\n",
        "!pip install --prefer-binary pathway"
      ],
      "execution_count": null,
      "outputs": [],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "id": "logging",
      "source": [
        "import logging\n",
        "\n",
        "logging.basicConfig(level=logging.CRITICAL)"
      ],
      "execution_count": null,
      "outputs": [],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "id": "4",
      "metadata": {},
      "source": [
        "# Evaluating RAG Applications with RAGAS"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5",
      "metadata": {},
      "source": [
        "Pathway streamlines the process of building RAG applications with always up-to-date knowledge. It empowers you to connect your LLM to live data sources and eliminates the need for separate ETL pipelines for knowledge management.\n",
        "\n",
        "However, simply building and deploying a RAG app isn't enough, and evaluations shouldn't be treated as an afterthought. In Pathway, we rely on frequent evaluation runs to keep our offerings reliable. This also prevents us from introducing any silent bugs into the pipeline.\n",
        "This guide offers a simplified look at how we evaluate our RAG solutions at Pathway. For a detailed view of the full pipeline, including additional evaluation components and logging, check out the  [complete CI workflow](https://github.com/pathwaycom/pathway/tree/main/integration_tests/rag_evals).\n",
        "\n",
        "You need to ensure that your RAG application delivers accurate and reliable results with YOUR data. This is where our blog post dives in. You will explore RAG evaluations, create synthetic test data if necessary, and learn how to optimize your Pathway RAG app.\n",
        "\n",
        "Here's a sneak peek of what we'll cover:\n",
        "\n",
        "- **Essential evaluation metrics**: We'll unpack key metrics used to assess different aspects of your RAG pipeline, including retrieval accuracy, generation quality, and overall system effectiveness.\n",
        "- **Creating synthetic dataset**: Build test data based on your files.\n",
        "- **Tweaking RAGAS to suit your needs**: From modifying metric calculations to modifying the LLM evaluator.\n",
        "- **Optimizing your Pathway RAG application**: Discover how to fine-tune your RAG system for optimal performance, tailored to your specific use case and dataset.\n",
        "\n",
        "\n",
        "## Table of contents\n",
        "- [Evaluation Metrics](#evaluation-metrics)\n",
        "- [Setup and Installation](#setup-and-installation)\n",
        "- [Dataset](#dataset)\n",
        "- [Synthetic Dataset Creation](#synthetic-dataset-creation)\n",
        "- [Launching the Pathway RAG App](#launching-the-pathway-rag-app)\n",
        "- [Evaluate with the Dataset](#evaluate-with-the-dataset)\n",
        "- [Improving-the-Accuracy:](#improving-the-accuracy)\n",
        "    - [Hybrid Index](#hybrid-index)\n",
        "    - [Parser](#using-a-different-parser)\n",
        "    - [Same Parser with Semantic Search](#lets-try-the-same-parser-with-the-semantic-search-retriever)\n",
        "    - [Changing the Embedder](#changing-the-embedder)\n",
        "    - [Changing the Prompt](#changing-the-prompt)\n",
        "- [Summary & Findings](#summary-findings)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6",
      "metadata": {},
      "source": [
        "## Evaluation Metrics\n",
        "\n",
        "RAG evaluation metrics can be categorized into two parts, `\"retrieval\"` metrics and `\"generation\"` metrics. Retrieval metrics are usually found in recommendation or information retrieval domains. Whereas generation metrics cover LLM related topics, including how the LLM makes use of the given context, hallucinations, truthfulness and so on.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7",
      "metadata": {},
      "source": [
        "Some of the retrieval metrics are:\n",
        "- `Hit@k`: Measures the proportion of times that the relevant item appears in the top-K retrieved results. This can be also mentioned as `\"Context Recall\"`, that is assuming there is only one relevant document.\n",
        "- `Context Recall`: Focuses on the comprehensiveness of the retrieved context, measuring the proportion of all relevant documents in the corpus that are successfully retrieved. It is formally defined as `(Number of Relevant Items Retrieved) / (Total Number of Relevant Items in Corpus)`. In simpler terms, recall tells you \"Of all the relevant documents that could have been retrieved, how many were actually retrieved?\". High recall signifies that your retrieval system is good at finding most of the relevant context available.\n",
        "- `Context Precision`: This metric focuses on the quality of the retrieved context by measuring the proportion of retrieved documents that are actually relevant to the query. Formally, it is calculated as `(Number of Relevant Items Retrieved) / (Total Number of Items Retrieved)`. In contrast to `Hit@k` (or \"Context Recall\") which emphasizes retrieving at least one relevant item within the top-K results, precision evaluates the relevance concentration within the retrieved set. Essentially, precision answers: \"Of all documents retrieved, how many were relevant?\".\n",
        "- `Mean Reciprocal Rank (MRR)`: Evaluates the ranking of retrieved documents by focusing on the position of the first relevant document in the ranked list. For each query, the Reciprocal Rank (RR) is calculated as 1 / rank, where rank is the position of the first relevant document. If there are no relevant documents in the retrieved list, RR is 0. MRR is then the mean of these reciprocal ranks across a set of queries. Generally, you shouldn't stress about this metric in your RAG application. This is largely because the benefit of having the most relevant context ranked at the top is less critical for LLMs.\n",
        "- `Normalized Discounted Cumulative Gain (NDCG)`: A ranking-based metric that evaluates the quality of retrieved results by considering both relevance and position. Unlike Hit@k and MRR, which primarily focus on whether relevant items appear at the top, NDCG assigns higher importance to highly relevant documents appearing earlier in the ranked list. This metric can be useful when you have more than one relevant items and their relevancy has float labels instead of booleans.\n",
        "\n",
        "As for the generation metrics:\n",
        "- `Faithfulness`: Evaluates how grounded the LLM's answer is in the retrieved context. It measures whether the claims in the generated answer are supported by the provided context. Penalizes the hallucinations.\n",
        "- `Answer correctness`: Factual correctness of the response. Even though this is under `generation` category, this metric generally covers the whole RAG pipeline.\n",
        "\n",
        "These are only small subset of the all metrics, however we found these set of metrics to be reliable indicators of the whole RAG application performance. If you are curious about the list of all available metrics in the RAGAS, check it out [here](https://docs.ragas.io/en/latest/concepts/metrics/available_metrics/)!"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8",
      "metadata": {},
      "source": [
        "## Setup and Installation"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9",
      "metadata": {},
      "source": [
        "Magic library is used for detecting file types in the `UnstructuredParser` module.\n",
        "\n",
        "If you are running this notebook on **MacOS**, you can install it with:\n",
        "> `brew install libmagic`\n",
        "\n",
        "If you are running the notebook on **colab** or any **linux** environment, you can install by running:\n",
        "> `apt install libmagic1`\n",
        "\n",
        "Install the rest of the dependencies:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "10",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "outputs": [],
      "source": [
        "pip install \"pathway[all]\"\n",
        "pip install ragas\n",
        "pip install langchain-openai"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "11",
      "metadata": {},
      "source": [
        "## Dataset\n",
        "\n",
        "Having a representative dataset is crucial for effective evaluations. It is recommended to set aside dedicated time to create a gold-standard dataset that accurately reflects your use case.\n",
        "\n",
        "\n",
        "To ensure robust evaluation, consider splitting your dataset into validation and test sets. Validation set helps fine-tune the retrieval and generation parameters, allowing for iterative improvements without overfitting to the final benchmark. Test set is kept separate from the tuning process, provides an unbiased measure of the performance, ensuring that optimizations generalize beyond the development phase."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "12",
      "metadata": {},
      "source": [
        "Here are the steps we will follow:\n",
        "- Prepare your docs to be in markdown format\n",
        "- Create and save synthetic dataset with RAGAS"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "13",
      "metadata": {},
      "source": [
        "## Synthetic Dataset Creation"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "14",
      "metadata": {},
      "source": [
        "### Prepare the documents as markdown"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "15",
      "metadata": {},
      "source": [
        "Here, we will use Pathway parsers to parse our document's content and save it as a markdown. Then, we will create a synthetic dataset based on the file contents with the `gpt-4o`. It is a good idea to create synthetic data with a model that is different than the one in your application. This is because LLM's bias will influence the wording, queries, and answers in your dataset. This may introduce unwanted bias in the metrics."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "16",
      "metadata": {},
      "outputs": [],
      "source": [
        "import os\n",
        "import getpass\n",
        "\n",
        "import pandas as pd\n",
        "import pathway as pw\n",
        "\n",
        "# To use advanced features with Pathway Scale, get your free license key from\n",
        "# https://pathway.com/features and paste it below.\n",
        "# To use Pathway Community, comment out the line below.\n",
        "pw.set_license_key(\"demo-license-key-with-telemetry\")\n",
        "\n",
        "\n",
        "from pathway.xpacks.llm import parsers\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "18",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "source": [
        "Define the helpers to save docs as markdown.\n",
        "This reads the file, parses it, and saves to specified folder with the same filename."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "19",
      "metadata": {},
      "outputs": [],
      "source": [
        "async def document_to_markdown(\n",
        "    input_path: str, output_folder: str, parser: pw.UDF = parsers.UnstructuredParser()\n",
        ") -> None:\n",
        "    os.makedirs(output_folder, exist_ok=True)\n",
        "\n",
        "    with open(input_path, \"rb\") as f:\n",
        "        file_bytes = f.read()\n",
        "        content = await parser.func(file_bytes)\n",
        "        file_md = \"\\n\\n\".join([split[0] for split in content])\n",
        "\n",
        "    md_file_name = os.path.splitext(os.path.basename(input_path))[0] + \".md\"\n",
        "\n",
        "    with open(f\"{output_folder}{os.path.sep}{md_file_name}\", \"w\") as f:\n",
        "        f.write(file_md)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "20",
      "metadata": {},
      "outputs": [],
      "source": [
        "MARKDOWN_FOLDER = \"./markdown_docs\"\n",
        "INPUT_FOLDER = \"./data\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "21",
      "metadata": {},
      "source": [
        "Download the Alphabet 10K report as an example PDF. Feel free to skip this step if you want to use your own documents. You will need to copy your documents to the `INPUT_FOLDER`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "22",
      "metadata": {},
      "outputs": [],
      "source": [
        "!wget -P \"$INPUT_FOLDER\" \"https://github.com/pathwaycom/llm-app/blob/main/templates/multimodal_rag/data/20230203_alphabet_10K.pdf\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "23",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "outputs": [],
      "source": [
        "await document_to_markdown(f\"{INPUT_FOLDER}/20230203_alphabet_10K.pdf\", MARKDOWN_FOLDER)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "24",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "source": [
        "### Configuring the Generations and Creating the Dataset\n",
        "\n",
        "Now we are finished with the parsing of the documents, let's create the synthetic dataset with RAGAS."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "25",
      "metadata": {},
      "outputs": [],
      "source": [
        "from langchain_community.document_loaders import DirectoryLoader"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "26",
      "metadata": {},
      "outputs": [],
      "source": [
        "loader = DirectoryLoader(MARKDOWN_FOLDER, glob=\"**/*.md\")\n",
        "docs = loader.load()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "27",
      "metadata": {},
      "outputs": [],
      "source": [
        "os.environ[\"OPENAI_API_KEY\"] = os.getenv(\"OPENAI_API_KEY\") or getpass.getpass(\n",
        "    \"Enter your OpenAI API key: \"\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "28",
      "metadata": {},
      "outputs": [],
      "source": [
        "from ragas.llms import LangchainLLMWrapper\n",
        "from ragas.embeddings import LangchainEmbeddingsWrapper\n",
        "from ragas.testset import TestsetGenerator\n",
        "from langchain_openai import ChatOpenAI\n",
        "from langchain_openai import OpenAIEmbeddings\n",
        "\n",
        "generator_llm = LangchainLLMWrapper(ChatOpenAI(model=\"gpt-4o\", temperature=0.0))\n",
        "generator_embeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings())\n",
        "\n",
        "generator = TestsetGenerator(\n",
        "    llm=generator_llm,\n",
        "    embedding_model=generator_embeddings,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "29",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "outputs": [],
      "source": [
        "# generate the dataset\n",
        "\n",
        "dataset = generator.generate_with_langchain_docs(\n",
        "    docs,\n",
        "    testset_size=20,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "30",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "source": [
        "Save the dataset into a file:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "31",
      "metadata": {},
      "outputs": [],
      "source": [
        "dataset.to_jsonl(\"synthetic_dataset.jsonl\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "32",
      "metadata": {},
      "outputs": [],
      "source": [
        "from ragas import EvaluationDataset, SingleTurnSample\n",
        "from ragas.testset.synthesizers.testset_schema import Testset"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "33",
      "metadata": {},
      "source": [
        "If you have a previously saved dataset, you can load it with `from_jsonl`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "34",
      "metadata": {},
      "outputs": [],
      "source": [
        "dataset = EvaluationDataset.from_jsonl(\"synthetic_dataset.jsonl\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "35",
      "metadata": {},
      "source": [
        "You may also download the synthetic dataset we created from the example file:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "36",
      "metadata": {},
      "outputs": [],
      "source": [
        "!wget -P \"$MARKDOWN_FOLDER\" \"https://gist.githubusercontent.com/berkecanrizai/4b036863a57cd6c93c7ca497c93abe2b/raw/4569e19bfd95fad05885fee32046e0b0d5d9d2cb/synthetic_dataset.jsonl\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "37",
      "metadata": {},
      "source": [
        "Have a peek at the dataset;"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "38",
      "metadata": {},
      "outputs": [],
      "source": [
        "dataset.to_pandas().head()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "39",
      "metadata": {},
      "source": [
        "> Notes on repeatability: The scores presented in this notebook are averaged over three independent runs to ensure reliability. Two of these runs used the provided synthetic dataset generated with the `UnstructuredParser`, while the third run utilized the data from the `PypdfParser`.\n",
        "\n",
        "> We found that LLM based evaluations can vary wildly between the runs. Also note that variables such as the order of the documents, wording of the answer & question pairs, and the LLM can have big impact on these scores. We also found that score variability and reliability is one of the main weaknesses of the `RAGAS`. We plan to repeat these experiments with [deepeval](https://github.com/confident-ai/deepeval) in the future, stay tuned!"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "40",
      "metadata": {},
      "source": [
        "## Launching the Pathway RAG App"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "41",
      "metadata": {},
      "source": [
        "Pathway [DocumentStore](https://pathway.com/developers/api-docs/pathway-xpacks-llm/document_store#pathway.xpacks.llm.document_store.DocumentStore) and [BaseRAGQuestionAnswerer](https://pathway.com/developers/api-docs/pathway-xpacks-llm/question_answering#pathway.xpacks.llm.question_answering.BaseRAGQuestionAnswerer) provides end to end solution for RAG applications.\n",
        "\n",
        "DocumentStore manages document ingestion from your data sources, as well as document processing that includes parsing, splitting, and the indexing.\n",
        "\n",
        "BaseRAGQuestionAnswerer creates a Pathway `RAG` application that:\n",
        "\n",
        "- Indexes the documents (via DocumentStore)\n",
        "- Exposes the question answering endpoints\n",
        "\n",
        "\n",
        "Let's keep things simple and test a naive RAG solution with the following components:\n",
        "- Unstructured Parser\n",
        "- Token based splitter\n",
        "- OpenAI embedder\n",
        "- Hybrid index that combines semantic search and keyword based BM25 search\n",
        "- A barebones RAG prompt\n",
        "\n",
        "For more information, check out the documentation:\n",
        "\n",
        "\n",
        "1. [Connectors](https://pathway.com/developers/user-guide/connect/pathway-connectors): Use Pathway\u2019s file reader to ingest the files.\n",
        "2. [Parsers](https://pathway.com/developers/api-docs/pathway-xpacks-llm/parsers): Utilize the UnstructuredParser to parse the documents. This parser supports multiple file types, including PDF, DOCX, and PPTX.\n",
        "3. [Text Splitters](https://pathway.com/developers/api-docs/pathway-xpacks-llm/splitters): Split the document content into chunks.\n",
        "4. [Embedders](https://pathway.com/developers/api-docs/pathway-xpacks-llm/embedders): Use OpenAI API for embeddings.\n",
        "5. [Vector/KNN Index](https://pathway.com/developers/api-docs/indexing#pathway.stdlib.indexing.BruteForceKnnFactory) (via BruteForceKnnFactory): Semantic index that is powered by an embedder.\n",
        "6. [BM25](https://pathway.com/developers/api-docs/indexing#pathway.stdlib.indexing.TantivyBM25) (via TantivyBM25Factory): Keyword based BM25 search.\n",
        "7. [HybridIndexFactory](https://pathway.com/developers/api-docs/indexing#pathway.stdlib.indexing.HybridIndexFactory): combines different indexes to build an hybrid index.\n",
        "8. [Prompts](https://pathway.com/developers/api-docs/pathway-xpacks-llm/prompts): Prompt template for RAG."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "42",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "outputs": [],
      "source": [
        "from pathway.stdlib.indexing import BruteForceKnnFactory, HybridIndexFactory\n",
        "from pathway.stdlib.indexing.bm25 import TantivyBM25Factory\n",
        "from pathway.udfs import DiskCache\n",
        "from pathway.xpacks.llm import embedders, llms, parsers, splitters\n",
        "from pathway.xpacks.llm.document_store import DocumentStore\n",
        "from pathway.xpacks.llm.question_answering import BaseRAGQuestionAnswerer, RAGClient\n",
        "from pathway.xpacks.llm.servers import QASummaryRestServer\n",
        "\n",
        "\n",
        "# read the text files under the data folder, we can also read from Google Drive, Sharepoint, etc.\n",
        "# See connectors documentation: https://pathway.com/developers/user-guide/connect/pathway-connectors to learn more\n",
        "folder = pw.io.fs.read(\n",
        "    path=INPUT_FOLDER,\n",
        "    format=\"binary\",\n",
        "    with_metadata=True,\n",
        ")\n",
        "\n",
        "# list of data sources to be indexed\n",
        "sources = [folder]\n",
        "\n",
        "# define the document processing steps\n",
        "parser = parsers.UnstructuredParser()\n",
        "\n",
        "text_splitter = splitters.TokenCountSplitter(min_tokens=150, max_tokens=450)\n",
        "\n",
        "embedder = embedders.OpenAIEmbedder(\n",
        "    cache_strategy=DiskCache(), retry_strategy=pw.udfs.ExponentialBackoffRetryStrategy()\n",
        ")\n",
        "\n",
        "index = BruteForceKnnFactory(embedder=embedder)\n",
        "\n",
        "llm = llms.OpenAIChat(model=\"gpt-4o\", cache_strategy=DiskCache())\n",
        "\n",
        "document_store = DocumentStore(\n",
        "    docs=sources, parser=parser, splitter=text_splitter, retriever_factory=index\n",
        ")\n",
        "\n",
        "prompt_template: str = \"\"\"You are an assistant for question-answering tasks. \\\n",
        "Use the following pieces of retrieved context to answer the question. \\\n",
        "If you don't know the answer, just say that you don't know.\n",
        "  Question: {query}\n",
        "\n",
        "  Context: {context}\n",
        "\n",
        "  Answer:\"\"\"\n",
        "\n",
        "# create the RAG app that will power the index, and serve the agent endpoint\n",
        "rag_app = BaseRAGQuestionAnswerer(\n",
        "    llm=llm,\n",
        "    indexer=document_store,\n",
        "    prompt_template=prompt_template,\n",
        "    search_topk=8,  # number of retrieved chunks for RAG\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "43",
      "metadata": {},
      "source": [
        "## Build and Run the Pathway server"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "44",
      "metadata": {},
      "outputs": [],
      "source": [
        "import multiprocessing\n",
        "\n",
        "\n",
        "# host and port of the RAG app\n",
        "pathway_host: str = \"0.0.0.0\"\n",
        "pathway_port: int = 8000"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "45",
      "metadata": {},
      "source": [
        "Once the app starts, it will:\n",
        "- Ingest your files\n",
        "- Parse and chunk the documents\n",
        "- Index the chunks\n",
        "- Host the RAG endpoint for question answering"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "46",
      "metadata": {},
      "outputs": [],
      "source": [
        "server = QASummaryRestServer(pathway_host, pathway_port, rag_app)\n",
        "\n",
        "server_process = multiprocessing.Process(target=server.run, kwargs=dict(threaded=False))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "47",
      "metadata": {},
      "source": [
        "Start the process:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "48",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "outputs": [],
      "source": [
        "server_process.start()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "49",
      "metadata": {},
      "source": [
        "RAGClient is the client that can query the Pathway RAG application.\n",
        "\n",
        "Let's test if the test files are indexed. This will list all the indexed documents in our Pathway server."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "50",
      "metadata": {},
      "outputs": [],
      "source": [
        "from pathway.xpacks.llm.question_answering import RAGClient\n",
        "\n",
        "pathway_client = RAGClient(pathway_host, pathway_port)\n",
        "pathway_client.list_documents()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "51",
      "metadata": {},
      "source": [
        "## Evaluate with the Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "52",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "source": [
        "Here, we will iterate over the samples and gather the RAG response and the context documents for each one of the test samples.\n",
        "\n",
        "Keep in mind that LLM evaluation metrics can fluctuate between runs.  Even minor details like context document order or wording can impact results. For more reliable testing, it's best to re-run tests multiple times and average the scores."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "53",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "outputs": [],
      "source": [
        "def predict_test_dataset(\n",
        "    dataset: Testset | EvaluationDataset, verbose: bool = True\n",
        ") -> EvaluationDataset:\n",
        "    predicted_samples: list[SingleTurnSample] = []\n",
        "\n",
        "    for sample in dataset.samples:\n",
        "        single_sample = sample.eval_sample if isinstance(dataset, Testset) else sample\n",
        "\n",
        "        if verbose:\n",
        "            print(f\"Predicting question: {single_sample.user_input}\")\n",
        "\n",
        "        pw_response: dict = pathway_client.answer(\n",
        "            prompt=single_sample.user_input, return_context_docs=True\n",
        "        )\n",
        "        resp: str = pw_response[\"response\"]\n",
        "        context_docs: list[str] = [elem[\"text\"] for elem in pw_response[\"context_docs\"]]\n",
        "\n",
        "        pred_sample = SingleTurnSample(\n",
        "            response=resp, retrieved_contexts=context_docs, **single_sample.to_dict()\n",
        "        )\n",
        "        predicted_samples.append(pred_sample)\n",
        "\n",
        "    return EvaluationDataset(samples=predicted_samples)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "54",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "outputs": [],
      "source": [
        "predicted_dataset = predict_test_dataset(dataset)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "55",
      "metadata": {},
      "outputs": [],
      "source": [
        "predicted_dataset.to_pandas().head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "56",
      "metadata": {},
      "outputs": [],
      "source": [
        "from ragas import evaluate\n",
        "\n",
        "\n",
        "from ragas.metrics import (\n",
        "    AnswerCorrectness,\n",
        "    Faithfulness,\n",
        "    context_recall,\n",
        "    context_precision,\n",
        ")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "57",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "source": [
        "Calculate the evaluation metrics with our selected metrics.\n",
        "\n",
        "We introduced few modifications on top of the default RAGAS settings, namely:\n",
        "- We completely ignored semantic similarity in the answer correctness, we found that it usually gives \"false positives\" and unnecessarily rewards bad predictions\\*.\n",
        "- We modified `answer_correctness_metric`'s prompt to be more forgiving and not look for the exact same words.\n",
        "- We increased `beta` parameter of the correctness to favor the recall rather than precision. We reward if LLM has more of relevant documents in the context. This is because LLM can choose to ignore irrelevant documents (False positive in context) which diminishes the importance of the precision.\n",
        "\n",
        "> \\* This issue stems from the limitations of commonly used encoder models, such as those generating sentence embeddings. These models are primarily trained on tasks like document similarity and natural language inference, making them effective at identifying semantically related text but not at evaluating factual accuracy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "58",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "outputs": [],
      "source": [
        "def run_ragas_evaluations(dataset: EvaluationDataset):\n",
        "\n",
        "    evaluator_llm = LangchainLLMWrapper(\n",
        "        ChatOpenAI(model=\"gpt-4o-mini\", temperature=0.0)\n",
        "    )\n",
        "\n",
        "    answer_correctness_metric = AnswerCorrectness(\n",
        "        llm=evaluator_llm,\n",
        "        weights=[\n",
        "            1.0,\n",
        "            0.0,\n",
        "        ],  # ignore the semantic similarity, it is often misleading. Prone to giving hi score to false labels.\n",
        "        max_retries=3,\n",
        "        beta=1.5,  # favor the recall a bit more\n",
        "    )\n",
        "\n",
        "    # adjust the evaluator LLM prompt to be more forgiving\n",
        "\n",
        "    correctness_prompt = answer_correctness_metric.get_prompts()[\"correctness_prompt\"]\n",
        "\n",
        "    correctness_prompt.instruction += \"\"\" Answer may be less or more verbose than the ground truth, that is fine.\n",
        "    If the ground truth is 'Yes' and answer is 'Yes, [... some details]', consider it as true.\"\"\"\n",
        "    answer_correctness_metric.set_prompts(**{\"correctness_prompt\": correctness_prompt})\n",
        "\n",
        "    metrics: list = [\n",
        "        answer_correctness_metric,\n",
        "        Faithfulness(llm=evaluator_llm),\n",
        "        context_recall,\n",
        "        context_precision,\n",
        "    ]\n",
        "    results = evaluate(dataset=dataset, metrics=metrics)\n",
        "    return results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "59",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "outputs": [],
      "source": [
        "ragas_evals_dataset = run_ragas_evaluations(predicted_dataset)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "60",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'answer_correctness': 0.5249, 'faithfulness': 0.6275, 'context_recall': 0.9353, 'context_precision': 0.7761}\n"
          ]
        }
      ],
      "source": [
        "ragas_evals_dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "61",
      "metadata": {},
      "source": [
        "\n",
        "We will learn how to improve these below."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "62",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "source": [
        "Let's inspect the eval metrics based on the questions. We see that in some cases, LLM had context recall of `1.0` but failed to answer the question correctly. This may be indicator of poor performance from the LLM, or irrelevant context (if precision is low) that caused LLM to be thrown off."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "63",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "outputs": [],
      "source": [
        "pd.DataFrame(ragas_evals_dataset.scores)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "64",
      "metadata": {},
      "source": [
        "Terminate the app;"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "65",
      "metadata": {},
      "outputs": [],
      "source": [
        "server_process.terminate()\n",
        "server_process.join()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "66",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "source": [
        "Clear the previous app from the Pathway engine"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "67",
      "metadata": {},
      "outputs": [],
      "source": [
        "def clear_pathway_graph() -> None:\n",
        "    from pathway.internals.parse_graph import G\n",
        "\n",
        "    G.clear()\n",
        "\n",
        "\n",
        "clear_pathway_graph()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "68",
      "metadata": {},
      "source": [
        "## Improving the Accuracy"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "69",
      "metadata": {},
      "source": [
        "\n",
        "A RAG application's performance is impacted by many variables, we can gather them under two broad categories that are linked together:\n",
        "- Retrieval\n",
        "- Generation\n",
        "\n",
        "Retrieval performance mainly consists of quality of the input data"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "70",
      "metadata": {},
      "source": [
        "### Hybrid Index\n",
        "\n",
        "Hybrid index combines semantic search and keyword based BM25 search.\n",
        "\n",
        "Pathway [HybridIndexFactory](https://pathway.com/developers/api-docs/indexing#pathway.stdlib.indexing.HybridIndexFactory) lets you combine different indexes to build an hybrid index:\n",
        "  1. [BM25](https://pathway.com/developers/api-docs/indexing#pathway.stdlib.indexing.TantivyBM25) (via TantivyBM25Factory) \u2192 Keyword based BM25 search.\n",
        "  2. [BruteForceKnn](https://pathway.com/developers/api-docs/indexing#pathway.stdlib.indexing.BruteForceKnn) \u2192 Vector-based semantic search"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "71",
      "metadata": {},
      "outputs": [],
      "source": [
        "folder = pw.io.fs.read(\n",
        "    path=INPUT_FOLDER,\n",
        "    format=\"binary\",\n",
        "    with_metadata=True,\n",
        ")\n",
        "\n",
        "sources = [folder]\n",
        "\n",
        "parser = parsers.UnstructuredParser()\n",
        "\n",
        "text_splitter = splitters.TokenCountSplitter(min_tokens=150, max_tokens=450)\n",
        "\n",
        "embedder = embedders.OpenAIEmbedder(\n",
        "    cache_strategy=DiskCache(), retry_strategy=pw.udfs.ExponentialBackoffRetryStrategy()\n",
        ")\n",
        "\n",
        "hybrid_index = HybridIndexFactory(\n",
        "    [\n",
        "        TantivyBM25Factory(),\n",
        "        BruteForceKnnFactory(embedder=embedder),\n",
        "    ]\n",
        ")\n",
        "\n",
        "llm = llms.OpenAIChat(model=\"gpt-4o\", cache_strategy=DiskCache())\n",
        "\n",
        "document_store = DocumentStore(\n",
        "    docs=sources, parser=parser, splitter=text_splitter, retriever_factory=hybrid_index\n",
        ")\n",
        "\n",
        "prompt_template: str = \"\"\"You are an assistant for question-answering tasks. \\\n",
        "Use the following pieces of retrieved context to answer the question. \\\n",
        "If you don't know the answer, just say that you don't know.\n",
        "  Question: {query}\n",
        "\n",
        "  Context: {context}\n",
        "\n",
        "  Answer:\"\"\"\n",
        "\n",
        "rag_app = BaseRAGQuestionAnswerer(\n",
        "    llm=llm,\n",
        "    indexer=document_store,\n",
        "    prompt_template=prompt_template,\n",
        ")\n",
        "\n",
        "\n",
        "server = QASummaryRestServer(pathway_host, pathway_port, rag_app)\n",
        "\n",
        "server_process = multiprocessing.Process(target=server.run, kwargs=dict(threaded=False))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "72",
      "metadata": {},
      "outputs": [],
      "source": [
        "server_process.start()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "73",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "outputs": [],
      "source": [
        "pathway_client = RAGClient(pathway_host, pathway_port)\n",
        "pathway_client.list_documents()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "74",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "outputs": [],
      "source": [
        "predicted_dataset_hybrid_index = predict_test_dataset(dataset)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "75",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "outputs": [],
      "source": [
        "ragas_evals_dataset_hybrid_index = run_ragas_evaluations(predicted_dataset_hybrid_index)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "76",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'answer_correctness': 0.5821, 'faithfulness': 0.5228, 'context_recall': 0.8966, 'context_precision': 0.8343}\n"
          ]
        }
      ],
      "source": [
        "ragas_evals_dataset_hybrid_index"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "77",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "source": [
        "We see that just by introducing hybrid retrieval, we improved the correctness metric by `10%`. Let's see if we can improve on that."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "78",
      "metadata": {},
      "outputs": [],
      "source": [
        "predicted_dataset_hybrid_index.to_pandas()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "79",
      "metadata": {},
      "outputs": [],
      "source": [
        "# terminate the Pathway app\n",
        "\n",
        "server_process.terminate()\n",
        "server_process.join()\n",
        "\n",
        "clear_pathway_graph()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "80",
      "metadata": {},
      "source": [
        "### Using a Different Parser\n",
        "\n",
        "Parsing is a crucial yet often overlooked component of RAG solutions. The quality of your retrieval depends heavily on how well your data is parsed\u2014garbage in, garbage out. A robust parser can significantly enhance your solution, while a poor one can break it. Pathway provides several ready-to-use parsers out of the box, see the [documentation](https://pathway.com/developers/api-docs/pathway-xpacks-llm/parsers). You also have the flexibility to develop and integrate custom parsers tailored to your specific needs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "81",
      "metadata": {},
      "outputs": [],
      "source": [
        "folder = pw.io.fs.read(\n",
        "    path=INPUT_FOLDER,\n",
        "    format=\"binary\",\n",
        "    with_metadata=True,\n",
        ")\n",
        "\n",
        "sources = [folder]\n",
        "\n",
        "parser = parsers.PypdfParser()\n",
        "\n",
        "text_splitter = None\n",
        "\n",
        "embedder = embedders.OpenAIEmbedder(\n",
        "    cache_strategy=DiskCache(), retry_strategy=pw.udfs.ExponentialBackoffRetryStrategy()\n",
        ")\n",
        "\n",
        "hybrid_index = HybridIndexFactory(\n",
        "    [\n",
        "        TantivyBM25Factory(),\n",
        "        BruteForceKnnFactory(embedder=embedder),\n",
        "    ]\n",
        ")\n",
        "\n",
        "llm = llms.OpenAIChat(model=\"gpt-4o\", cache_strategy=DiskCache())\n",
        "\n",
        "document_store = DocumentStore(\n",
        "    docs=sources, parser=parser, splitter=text_splitter, retriever_factory=hybrid_index\n",
        ")\n",
        "\n",
        "prompt_template: str = \"\"\"You are an assistant for question-answering tasks. \\\n",
        "Use the following pieces of retrieved context to answer the question. \\\n",
        "If you don't know the answer, just say that you don't know.\n",
        "  Question: {query}\n",
        "\n",
        "  Context: {context}\n",
        "\n",
        "  Answer:\"\"\"\n",
        "\n",
        "rag_app = BaseRAGQuestionAnswerer(\n",
        "    llm=llm,\n",
        "    indexer=document_store,\n",
        "    prompt_template=prompt_template,\n",
        ")\n",
        "\n",
        "\n",
        "server = QASummaryRestServer(pathway_host, pathway_port, rag_app)\n",
        "\n",
        "server_process = multiprocessing.Process(target=server.run, kwargs=dict(threaded=False))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "82",
      "metadata": {},
      "outputs": [],
      "source": [
        "server_process.start()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "83",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "outputs": [],
      "source": [
        "pathway_client = RAGClient(pathway_host, pathway_port)\n",
        "pathway_client.list_documents()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "84",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "outputs": [],
      "source": [
        "predicted_dataset_pypdf_parser = predict_test_dataset(dataset)\n",
        "ragas_evals_dataset_pypdf_parser = run_ragas_evaluations(predicted_dataset_pypdf_parser)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "85",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'answer_correctness': 0.6896, 'faithfulness': 0.6609, 'context_recall': 0.9088, 'context_precision': 0.8035}\n"
          ]
        }
      ],
      "source": [
        "ragas_evals_dataset_pypdf_parser"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "86",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "source": [
        "This had quite an impact! We managed to improve our last best score by more than 30%."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "87",
      "metadata": {},
      "outputs": [],
      "source": [
        "# terminate the Pathway app\n",
        "\n",
        "server_process.terminate()\n",
        "server_process.join()\n",
        "\n",
        "clear_pathway_graph()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "88",
      "metadata": {},
      "source": [
        "### Let's Try the Same Parser with the Semantic Search Retriever"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "89",
      "metadata": {},
      "outputs": [],
      "source": [
        "folder = pw.io.fs.read(\n",
        "    path=INPUT_FOLDER,\n",
        "    format=\"binary\",\n",
        "    with_metadata=True,\n",
        ")\n",
        "\n",
        "sources = [folder]\n",
        "\n",
        "parser = parsers.PypdfParser()\n",
        "# pypdf parser splits documents by the page, so we don't need another splitter\n",
        "text_splitter = None\n",
        "\n",
        "embedder = embedders.OpenAIEmbedder(\n",
        "    cache_strategy=DiskCache(), retry_strategy=pw.udfs.ExponentialBackoffRetryStrategy()\n",
        ")\n",
        "\n",
        "llm = llms.OpenAIChat(model=\"gpt-4o\", cache_strategy=DiskCache())\n",
        "\n",
        "document_store = DocumentStore(\n",
        "    docs=sources,\n",
        "    parser=parser,\n",
        "    splitter=text_splitter,\n",
        "    retriever_factory=BruteForceKnnFactory(embedder=embedder),\n",
        ")\n",
        "\n",
        "prompt_template: str = \"\"\"You are an assistant for question-answering tasks. \\\n",
        "Use the following pieces of retrieved context to answer the question. \\\n",
        "If you don't know the answer, just say that you don't know.\n",
        "  Question: {query}\n",
        "\n",
        "  Context: {context}\n",
        "\n",
        "  Answer:\"\"\"\n",
        "\n",
        "rag_app = BaseRAGQuestionAnswerer(\n",
        "    llm=llm,\n",
        "    indexer=document_store,\n",
        "    prompt_template=prompt_template,\n",
        ")\n",
        "\n",
        "\n",
        "server = QASummaryRestServer(pathway_host, pathway_port, rag_app)\n",
        "\n",
        "server_process = multiprocessing.Process(target=server.run, kwargs=dict(threaded=False))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "90",
      "metadata": {},
      "outputs": [],
      "source": [
        "server_process.start()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "91",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "outputs": [],
      "source": [
        "predicted_dataset_semantic = predict_test_dataset(dataset)\n",
        "ragas_evals_dataset_semantic = run_ragas_evaluations(predicted_dataset_semantic)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "92",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'answer_correctness': 0.7026, 'faithfulness': 0.6485, 'context_recall': 0.9382, 'context_precision': 0.7915}\n"
          ]
        }
      ],
      "source": [
        "ragas_evals_dataset_semantic"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "93",
      "metadata": {},
      "source": [
        "We see that correctness and recall both increased slightly."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "94",
      "metadata": {},
      "outputs": [],
      "source": [
        "# terminate the Pathway app\n",
        "\n",
        "server_process.terminate()\n",
        "server_process.join()\n",
        "\n",
        "clear_pathway_graph()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "95",
      "metadata": {},
      "source": [
        "### Changing the Embedder"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "96",
      "metadata": {},
      "source": [
        "Up until now, we had been using the OpenAI's `text-embedding-ada-002` as the embedding model. You may also try the larger and better `text-embedding-3-large` or the cheaper and smaller `text-embedding-3-small` models.\n",
        "\n",
        "Although API based embedders are well performing and easy to use, if you are concerned about data privacy, you will need a locally hosted embedder. Pathway enables you to use local & open-source models through [embedders.SentenceTransformerEmbedder](https://pathway.com/developers/api-docs/pathway-xpacks-llm/embedders#pathway.xpacks.llm.embedders.SentenceTransformerEmbedder).\n",
        "\n",
        "Some of the popular open-source embedders are `gte-large-en-v1.5`, `bge-m3`, and `nomic-embed-text-v1.5`. We found that `gte-large-en-v1.5` usually produces good results, so let's try swapping the embedder with it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "97",
      "metadata": {},
      "outputs": [],
      "source": [
        "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "98",
      "metadata": {},
      "outputs": [],
      "source": [
        "folder = pw.io.fs.read(\n",
        "    path=INPUT_FOLDER,\n",
        "    format=\"binary\",\n",
        "    with_metadata=True,\n",
        ")\n",
        "\n",
        "sources = [folder]\n",
        "\n",
        "parser = parsers.PypdfParser()\n",
        "# pypdf parser splits documents by the page, so we don't need another splitter\n",
        "text_splitter = None\n",
        "\n",
        "embedder = embedders.SentenceTransformerEmbedder(\n",
        "    model=\"Alibaba-NLP/gte-large-en-v1.5\",\n",
        "    call_kwargs={\"show_progress_bar\": False},\n",
        "    trust_remote_code=True,\n",
        ")\n",
        "\n",
        "llm = llms.OpenAIChat(model=\"gpt-4o\", cache_strategy=DiskCache())\n",
        "\n",
        "document_store = DocumentStore(\n",
        "    docs=sources,\n",
        "    parser=parser,\n",
        "    splitter=text_splitter,\n",
        "    retriever_factory=BruteForceKnnFactory(embedder=embedder),\n",
        ")\n",
        "\n",
        "prompt_template: str = \"\"\"You are an assistant for question-answering tasks. \\\n",
        "Use the following pieces of retrieved context to answer the question. \\\n",
        "If you don't know the answer, just say that you don't know.\n",
        "  Question: {query}\n",
        "\n",
        "  Context: {context}\n",
        "\n",
        "  Answer:\"\"\"\n",
        "\n",
        "rag_app = BaseRAGQuestionAnswerer(\n",
        "    llm=llm,\n",
        "    indexer=document_store,\n",
        "    prompt_template=prompt_template,\n",
        ")\n",
        "\n",
        "\n",
        "server = QASummaryRestServer(pathway_host, pathway_port, rag_app)\n",
        "\n",
        "server_process = multiprocessing.Process(target=server.run, kwargs=dict(threaded=False))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "99",
      "metadata": {},
      "outputs": [],
      "source": [
        "server_process.start()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "100",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "outputs": [],
      "source": [
        "predicted_dataset_gtembedder = predict_test_dataset(dataset)\n",
        "ragas_evals_dataset_gtembedder = run_ragas_evaluations(predicted_dataset_gtembedder)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "101",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'answer_correctness': 0.641, 'faithfulness': 0.5327, 'context_recall': 0.9357, 'context_precision': 0.8268}\n"
          ]
        }
      ],
      "source": [
        "ragas_evals_dataset_gtembedder"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "102",
      "metadata": {},
      "source": [
        "Hmm, seems like this embedder cannot quite work as well as the previous one. Weirdly, faithfulness score dropped significantly, maybe the ordering of the chunks is the reason.\n",
        "\n",
        "Let's see if we can improve the performance with the prompt."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "103",
      "metadata": {},
      "outputs": [],
      "source": [
        "# terminate the Pathway app\n",
        "\n",
        "server_process.terminate()\n",
        "server_process.join()\n",
        "\n",
        "clear_pathway_graph()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "104",
      "metadata": {},
      "source": [
        "### Changing the Prompt"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "105",
      "metadata": {},
      "source": [
        "Prompt is one of the more important aspects of a good RAG solution. Although we will aim for a prompt that will work in general, you may want to modify your prompt in consideration with your users' expectations, business goals, domain knowledge or other variables.\n",
        "\n",
        "Now, let's keep the same embedder as above and change the prompt to be a bit more compute intensive."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "106",
      "metadata": {},
      "outputs": [],
      "source": [
        "folder = pw.io.fs.read(\n",
        "    path=INPUT_FOLDER,\n",
        "    format=\"binary\",\n",
        "    with_metadata=True,\n",
        ")\n",
        "\n",
        "sources = [folder]\n",
        "\n",
        "parser = parsers.PypdfParser()\n",
        "# pypdf parser splits documents by the page, so we don't need another splitter\n",
        "text_splitter = None\n",
        "\n",
        "embedder = embedders.SentenceTransformerEmbedder(\n",
        "    model=\"Alibaba-NLP/gte-large-en-v1.5\",\n",
        "    call_kwargs={\"show_progress_bar\": False},\n",
        "    trust_remote_code=True,\n",
        ")\n",
        "\n",
        "llm = llms.OpenAIChat(model=\"gpt-4o\", cache_strategy=DiskCache())\n",
        "\n",
        "document_store = DocumentStore(\n",
        "    docs=sources,\n",
        "    parser=parser,\n",
        "    splitter=text_splitter,\n",
        "    retriever_factory=BruteForceKnnFactory(embedder=embedder),\n",
        ")\n",
        "\n",
        "prompt_template: str = \"\"\"You are an assistant for question-answering tasks. \\\n",
        "Use the following pieces of retrieved context to answer the question. \\\n",
        "\n",
        "Before answering the question, first think about and list the relevant parts from the given context. \\\n",
        "Then, answer the question based on the facts you have listed.\n",
        "\n",
        "Always structure your responses in the following format:\n",
        "Relevant contexts: [Write the relevant parts of the context for given question]\n",
        "Answer: [Detailed reponse to the user's question that is grounded by the facts you listed]\n",
        "\n",
        "If you don't know the answer, just say that you don't know.\n",
        "\n",
        "\n",
        "Question: {query}\n",
        "\n",
        "Context: {context}\n",
        "\n",
        "Answer:\"\"\"\n",
        "\n",
        "rag_app = BaseRAGQuestionAnswerer(\n",
        "    llm=llm,\n",
        "    indexer=document_store,\n",
        "    prompt_template=prompt_template,\n",
        ")\n",
        "\n",
        "\n",
        "server = QASummaryRestServer(pathway_host, pathway_port, rag_app)\n",
        "\n",
        "server_process = multiprocessing.Process(target=server.run, kwargs=dict(threaded=False))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "107",
      "metadata": {},
      "outputs": [],
      "source": [
        "server_process.start()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "108",
      "metadata": {},
      "outputs": [],
      "source": [
        "predicted_dataset_gtembedder_semantic = predict_test_dataset(dataset)\n",
        "ragas_evals_dataset_gtembedder_semantic = run_ragas_evaluations(\n",
        "    predicted_dataset_gtembedder_semantic\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "109",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'answer_correctness': 0.7448, 'faithfulness': 0.8011, 'context_recall': 0.9471, 'context_precision': 0.8153}\n"
          ]
        }
      ],
      "source": [
        "ragas_evals_dataset_gtembedder_semantic"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "110",
      "metadata": {},
      "source": [
        "We can see some clear improvements in terms of correctness. As expected, retrieval metrics remained unchanged from the previous runs (except the one with different embedder)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "111",
      "metadata": {},
      "outputs": [],
      "source": [
        "# terminate the Pathway app\n",
        "\n",
        "server_process.terminate()\n",
        "server_process.join()\n",
        "\n",
        "clear_pathway_graph()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "112",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "source": [
        "## Summary & Findings"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "113",
      "metadata": {},
      "source": [
        "There is no \"one size fits all\" logic when it comes to RAG. You need to find what suits you best and start working from there.\n",
        "\n",
        "Pathway allows you to build RAG applications that are always live & up-to-date and available. Whether you are building a financial analysis tool for yourself or an internal application for lawyers, you need to think about how to update and refresh your knowledge base. With the help of dynamic connectors, you can focus on problems that matter.\n",
        "\n",
        "We also learned that we can improve our performance by using a better parsing strategy, increasing the amount of retrieved chunks, or introducing hybrid retrieval rather then semantic search strategy.\n",
        "\n",
        "However, we have only explored a single RAG paradigm that only consisted of simple retrieval and generation. There are many more ways that are left to explore!\n",
        "For instance, knowledge-graphs can help the LLM to have more relevant context, or agent-driven architectures can unlock new search & retrieval capabilities that can adapt, re-try or reason before taking actions. Pathway can help you build such applications as well.\n",
        "\n",
        "If you are interested in agents, you may explore our [LangGraph RAG agents with Pathway cookbook](https://github.com/pathwaycom/llm-app/tree/main/cookbooks/self-rag-agents) and stay tuned for more content!\n",
        "\n",
        "\n",
        "If you are interested in diving deeper into the topic, here are some good references to get started:\n",
        "\n",
        "- https://pathway.com/developers/templates\n",
        "- https://docs.ragas.io/en/latest/concepts/metrics/overview/#different-types-of-metrics\n",
        "- https://github.com/beir-cellar/beir\n",
        "- https://huggingface.co/spaces/mteb/leaderboard"
      ]
    }
  ],
  "metadata": {
    "jupytext": {
      "cell_metadata_filter": "-all",
      "main_language": "python",
      "notebook_metadata_filter": "-all"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.14"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}